Method and apparatus for scoring electronic documents

ABSTRACT

A method, apparatus and data structure is provided to determine a score for an electronic document, such as a webpage, image, audio recording, video recording or other electronic content, to aid in the ranking and retrieval of the electronic document. The score for an electronic document is based on weighted subjective user ratings of the electronic document by members of a member set. Weight factors are assigned to the members of the member set who then rate the electronic document. The score is determined for the electronic document based on the ratings of the electronic document by the members in the member set where each member&#39;s rating is weighted by a weight factor specific for the member who has provided the rating. The weight factor for a member is based on ratings assigned to that member by other members in the member set.

This invention is in the field of information retrieval systems and inparticular systems and methods for providing a rating of search resultsand other electronic documents.

BACKGROUND

The World Wide Web provides a large collection of interconnected contentin the form of electronic documents, images and other media content.Over the years the web has grown to an immense size and containswebpages and other content on just about every subject a person couldthink of. As a result of its growth to such an immense size, locatingcontent on the web has become of primary concern with the result thatthere are numerous search services now available.

Many of these search services take the form of a search engine, where auser can input a search query in the form of one or more search termswith connectors placed in between the terms. The search engine thentakes the search query and attempts to match it to webpages on the webthat have been indexed by the search engine. By matching the searchquery to a number of different webpages, the search engine generates alist of search results and returns the list of search results to theuser. Each search result in the list references the located webpage orother electronic document and typically includes a link that a user canuse to access the located webpage or other located electronic document.

The search engines typically locate what they consider to be “relevant”webpages or electronic documents by using specially created indexesand/or databases where the relevancy of a document identified in anindex or database is based on terms from the search query being present.The located documents are then further ranked so that the “best” resultsappear higher in the list of search results and the “poorer” resultsappear closer to the bottom or end of the list of search results.

Additionally, it has become a fairly big business to consult on websitedesign in order to use tricks and loopholes in the more commonalgorithms used by search engines to have a webpage ranked higher insearch results than another webpage which might be as good qualitativelyif not better than the higher ranked webpage.

The ranking of the located search results is typically done usingalgorithms that often base the ranking on how closely the search querymatches the located webpages (usually on how the webpage is described inthe search engine's index or database) and other criteria. In somecases, because the search engines only receive a search query containingsearch terms, the ranking of the webpages located by a search engine canbe heavily based on the occurrence of the search terms in the index ordatabase identifying the webpage, however, other factors can also betaken into account, such as whether the domain name matches the searchquery or whether a webpage is a sponsored link that has paid the searchengine to be ranked higher.

While many of these algorithms may be good at ranking located webpagesby the criteria of “relevancy” used by the algorithms, this ranking isbased on objective factors. They are typically unable to determine whichof the located webpages may be qualitatively “better” than other locatedwebpages, which is often a subjective quality assessment that cannot beassessed on a purely objective basis. By relying on objectively definedparameters such as the number of times a search term appears on awebpage or whether the domain name contains one or more of the terms inthe search query these algorithms fail to provide rankings of thelocated webpages based on the subjective quality of a webpage. Often,even though a webpage may use commonly used search terms and thereforetypically rank quite highly in a list of search results, the overallquality of the webpage may not be that high or as good as another sitethat does not use the search terms as frequently.

While many search engines do not even attempt to address howqualitatively good search results may be, some search engines do usealgorithms that attempt to determine which search results arequalitatively “better” than other search results. One example of this isthe algorithm disclosed by U.S. Pat. No. 6,285,999 to Page that uses thenumber of links between webpages to try to assess the quality of awebpage. The algorithm is based on the underlying theory that websitesthat are linked to by a relatively large number of other unrelatedwebsites are more likely to be qualitatively “better” than websites thathave few other websites linking to it. Even in trying to determine howsubjectively “good” a website might be, this algorithm is still limitedto using objectively measurable factors (in this case the amount oflinks) to attempt to approximate how subjectively “good” a webpage maybe.

There is a need to provide some type of rating of the quality of awebpage or other content; a rating that reflects how “good” the webpageor other content is.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a method andapparatus that overcomes problems in the prior art.

In a first aspect, a method for assigning a score to an electronicdocument is provided. The method comprises: assigning a weight factor toeach member of a plurality of members; receiving a rating value of anelectronic document from each of at least two of the members of theplurality of members; and determining a score for the electronicdocument, the score based on each rating value assigned to theelectronic document by a member with each rating value being weighted bythe weight factor assigned to the member who provided the rating.

In as second aspect, a computer readable memory having recorded thereonstatements and instructions for execution by a data processing system tocarry out the method of assigning a score to an electronic document isprovided. The method comprises: assigning a weight factor to each memberof a plurality of members; receiving a rating value of an electronicdocument from each of at least two of the members of the plurality ofmembers; and determining a score for the electronic document, the scorebased on each rating value assigned to the electronic document by amember with each rating value being weighted by the weight factorassigned to the member who provided the rating.

In a third aspect, a data processing system for assigning a score to anelectronic document is provided. The data processing system comprises:at least one processor; a memory operatively coupled to the at least oneprocessor; and a program module stored in the memory and operative forproviding instructions to the at least one processor, the at least oneprocessor responsive to the instructions of the program module. Theprogram module is operative for: assigning a weight factor to eachmember of a plurality of members; receiving a rating value of anelectronic document from each of at least two of the members of theplurality of members; and determining a score for the electronicdocument, the score based on each rating value assigned to theelectronic document by a member with each rating value being weighted bythe weight factor assigned to the member who provided the rating.

In a fourth aspect, a memory for storing data for access by at least oneapplication program being executed on a data processing system isprovided. The memory comprises a data structure stored in said memory,said data structure including information resident in a database used bysaid at least one application program. The data structure comprising: adocument record associated with an electronic document and having ascore value; and at least two member records each member recordrepresenting one of a plurality of members, each member record having aweight factor and associated with the document record by a rating valueassigned to the electronic document by the member represented by themember record. Wherein the score value of the document record is basedon each rating value assigned to the electronic document associated withthe document record and with each rating value weighted by the weightfactor of the member record associated with the rating value.

Online communities in the form of social networks have become popular onthe interne. Online social networks allow members to interact and makeconnections with other members in the online community; either byjoining them together in subgroups or connecting members directly aspeers. Common examples of these online social networks includeFriendster™ and MySpace™, which allow users to link to other users,share information about themselves and send messages to each other.These existing online social networks are primarily focused on membersmeeting other members through linking to pages the members create,bulletin boards or direct messaging services. Often these sites arefocused on geographical locations so that people online can meet peoplewho live near them.

By structuring an online social network so that the members of thesocial network can rate electronic documents such as webpages and otherelectronic content, a subjective rating of an electronic document can beobtained. Because these members form an online community, the onlinecommunity can be structured so that not only can the members rateelectronic documents, but the members in the online network can alsorank other members. In this manner, not only can subjective ratings ofelectronic documents such as webpages or other electronic content begiven a weighted rating, based on a subjective quality of how good anelectronic document is, in the opinions of the members in the onlinecommunity, but the members ratings of the electronic document can befurther weighted based on how the online community views the members whoare doing the rating. A member that is highly rated by other members inthe online community will have more weight placed on his or her ratingsof an electronic document than another member that is rated much lowerby the online community.

Rather than attempting to indirectly approximate the subjective qualityof electronic document, such as a webpage, by looking at objectivelymeasurable attributes that a computer system can measure and evaluate,the present apparatus and methods allow a rating to be determined forelectronic content that is directly based on the subject qualityassessment of the electronic content by an online community.

DESCRIPTION OF THE DRAWINGS

While the invention is claimed in the concluding portions hereof,preferred embodiments are provided in the accompanying detaileddescription which may be best understood in conjunction with theaccompanying diagrams where like parts in each of the several diagramsare labeled with like numbers, and where:

FIG. 1 is schematic illustration of a conventional data processingsystem capable of implementing the methods of the present invention;

FIG. 2 is schematic illustration of a network configuration wherein adata processing system operative to implement the provided methods inaccordance with the present invention is connected over a network to aplurality of servers operating as a search engine;

FIG. 3 is a schematic illustration of a member record;

FIG. 4 is a schematic illustration of a rating record;

FIG. 5 is a flowchart of a method for a user to rate an electronicdocument the user has reviewed;

FIG. 6 is a flowchart of a method for a user to rate and categorize anelectronic document the user has reviewed;

FIG. 7 is a screen shot of an exemplary browser for ranking a webpagebeing viewed;

FIG. 8 is a flowchart of a method for a user to rank another member inthe membership set;

FIG. 9 is a flowchart of a method for setting the weights of the variousmembers in a member set; and

FIG. 10 is a flowchart of a method for calculating a score values forelectronic documents based on the weighted ratings of the electronicdocuments.

DETAILED DESCRIPTION OF THE ILLUSTRATED EMBODIMENTS

FIG. 1 illustrates a data processing system 101 suitable for supportingthe operation of the present invention. The data processing system 101could be a personal computer, workstation, server, mobile computingdevice, cell phone, etc. The data processing system 101 typicallycomprises: at least one processing unit 103; a memory storage device104; at least one input device 105; a display device 106; a programmodule 108 and a network interface 110.

The processing unit 103 can be any processor that is typically known inthe art with the capacity to run the provided methods and is operativelycoupled to the memory storage device 4 through a system bus. In somecircumstances the data processing system 101 may contain more than oneprocessing unit 103. The memory storage device 104 is operative to storedata and can be any storage device that is known in the art, such as alocal hard-disk, etc. and can include local memory employed duringactual execution of the program code, bulk storage, and cache memoriesfor providing temporary storage. Additionally, the memory storage device104 can be a database that is external to the data processing system 101but operatively coupled to the data processing system 101. The inputdevice 105 can be any suitable device suitable for inputting data intothe data processing system 101, such as a keyboard, mouse or data portsuch as a network connection and is operatively coupled to theprocessing unit 103 and operative to allow the processing unit 103 toreceive information from the input device 105. The display device 106 isa CRT, LCD monitor, etc. operatively coupled to the data processingsystem 101 and operative to display information. The display device 106could be a stand-alone screen or if the data processing system 101 is amobile device, the display device 106 could be integrated into a casingcontaining the processing unit 103 and the memory storage device 104.The program module 108 is stored in the memory storage device 104 andoperative to provide instructions to processing unit 103 and theprocessing unit 103 is responsive to the instructions from the programmodule 108.

The network interface 110 allows the data processing system 101 to beconnected to a computer network such as an intranet or the internet.This network interface 110 could be an Ethernet card, modem or otherline based network system or a wireless connection such as CDPD,Bluetooth, 802.11, or other suitable network.

Although other internal components of the data processing system 101 arenot illustrated, it will be understood by those of ordinary skill in theart that only the components of the data processing system 101 necessaryfor an understanding of the present invention are illustrated and thatmany more components and interconnections between them are well knownand can be used.

Additionally, the invention can take the form of a computer readablemedium having recorded thereon statements and instructions for executionby a data processing system 101. For the purposes of this description, acomputer readable medium can be any apparatus that can contain, store,communicate, propagate, or transport the program for use by or inconnection with the instruction execution system, apparatus, or device.

Overview of System

Data processing system 101 typically runs a browsing application, suchas Microsoft's Internet Explorer™, Mozilla Firefox™, Apple Safari™,Netscape Navigator™, Opera™ or other browser program. In one aspect, aplugin is used with the browsing application to implement some of thedisclosed methods.

FIG. 2 illustrates a network configuration wherein the data processingsystem 101 is connected over a network 255 to at least one search engineserver 250. The network 255 can comprise a single network such as theinternet or it could comprise a plurality of networks such as a wirelessnetwork, a wired network, etc. While in one aspect, the system issuitable for use with the Internet, it should be understood that thenetwork 255 could comprise other types of networks, such as an intranetor other suitable network.

A plurality of content servers 265 ₁ to 265 _(N) are configured to actas web servers and provide data and electronic content, generallyalthough not necessarily in the form of websites containing webpages, tothe data processing system 101. The data processing system 101 canaccess any of the content servers 265 to view electronic documents, suchas webpages, contained on the content servers 265. Typically, the dataprocessing system 101 uses a web browser application to access any ofthe content servers 265 ₁ to 265 _(N), which are web servers and theelectronic documents accessed on any of the content servers 265 aregenerally files in a markup language which the browser displays as a website and web pages on the data processing system 101.

A search engine server 250 is provided, configured to provide searchresult data to the data processing system 101. Well-known search enginesinclude Google™, Yahoo!Search™, MSN Search™, Ask.com™, but there aremany other search engines and many may be sufficient for search engineserver 250. Typically, a search query transmitted to the search engineserver 250 from the data processing system 101 is used by the searchengine server 250 to locate a plurality of electronic documents on thecontent servers 265. For example, the search results my be a list ofelectronic documents located by the search engine 250 that resides onthe content servers 265 with links to the content servers 265 to allow auser using the data processing system 101 to access the locatedelectronic document. Typically, the search engine server 250 accessesvarious search indexes that are populated with: links to electronicdocuments; meta-data describing the content of indexed electronicdocuments; or other meta-data and uses these electronic document indexesto construct a list of search results, as is known in the art. As iscommon in the art, these electronic document indexes are typicallypopulated using a web crawler.

Once the search engine server 250 has located a number of searchresults, a list of search results is passed back to the data processingsystem 101 to the user. Generally although not necessarily; each searchresult in the list of search results comprises a link to an electronicdocument located on one of the content servers 265.

A central server 260 is provided and is operatively connected to a scoredatabase 270. The score database 270 contains a plurality of memberrecords 280 and a plurality of rating records 290.

A number of users (or members) form a member set that has access to thecentral server 260. Each user in the set of members will have a memberrecord 280 associated with him or her and stored in the score database270. In one aspect, member record 280 is in the format of a memberobject 300 as shown in FIG. 3. Each member object 300 corresponds withone of the members in the member set and is the record used by themember as they are accessing the central server 260. Each member object300 comprises: a user field 310 that contains an identifier (i.e. thename or a user name of the user) which identifies the specific user ormember that corresponds to the member object 300; a user password field320 that contains the password of the user or member; a weight field330, that indicates a weighting factor to be applied to any rating madeby the user; and an avatar field 340 that contains a link to a graphicthe member is using for his or her avatar.

Each member object 300 can be associated with one or more other memberobjects 300. This occurs when one member rates another member in themember set. The association is defined by a user rating object 350 thathas a user rating field 360 which contains the rating that the onemember has given to the other member. In this manner, a member can rateother members in the member set and this rating will become a factor inthat other member's rating contained in the weight field 330 of themember object 300 corresponding to the other member.

In addition to the score database 270 storing a number of member records280, the score database 270 also stores a number of document records290. In one aspect, document record 290 is in the format shown in FIG.4. The document record 290 contains a document object 400 associatedwith zero (0) or more document rating objects 405. Each document object400 corresponds to an electronic document. The electronic document couldbe a webpage, website, graphic image, sound recording movie recording orother electronic content located on a content server 265 and eachdocument rating object 405 associated with the document object 400corresponds to a rating of the electronic document by a member in themember set.

Typically, each document object 400 comprises: a title field 410,identifying the title of the electronic document; a graphicalrepresentation 415 that provides a graphical representation of theelectronic document corresponding to the document object 400; an addressfield 420 that identifies the location of the electronic document on thecontent servers 265 typically using the uniform resource locator address(URL address) or the uniform resource identifier address (URI address);optionally, a category field 460, that identifies a category that theelectronic document has been classed in; and a score field 470, thatcontains a score value determined for the electronic document and basedon how members in the member set have rated the electronic document.

Additionally, each document object 400 is associated with one or moredocument rating objects 405 where each document rating object 405 holdsa rating assigned to the page or other piece of content by a member inthe member set. Each document rating object 405 typically comprises: adocument rating field 440, which contains a rating of the electronicdocument made by a member; a comment field 450, containing any comments(typically in a text string) that the member might have made regardingthe electronic content; and a date stamp 455, indicating the date and/ortime the member rated the electronic document.

The document rating field 440 is used to store a rating of theelectronic document by the member in the associated member object 300.This document rating field 440 reflects the associated member's opinionor judgment of the electronic document. The document rating field 440typically comprises a rating from 1 to 10 with 10 being the highest and1 being the lowest, however, any sufficient rating system could be used,such as numeric, alphanumeric (i.e. excellent, good, average, poor,horrible), symbolic (i.e. four stars, two thumbs up) or any othersuitable rating system.

The comment field 450 can hold a comment made by the associated memberregarding the electronic document identified in the address field 420 ofthe associated document object 400. The comment can be a long string oftext containing a comment about the site made by the creator.

The score field 470 in the document object 400 shows a rating of theelectronic document identified by the address field 420. The ratingcontained in the score field 470 is based on the rating contained in thecontent rating field 440 of all of the associated document ratingobjects 405 with each rating being further weighted with the weightcontained in the weight field 330 of the member object 300 associatedwith the member who provided the rating in the document rating field440. In this manner, an electronic document can be rated by a number ofmembers with each rating assigned by a member being further weightedbased on how other members have personally rated that member.

It is to be understood that the member record data object 300 and thedocument object 400 and document rating object 405 are only illustrativeand that other formats with different fields may be used for the memberrecords 280 and document records 290 contained in the score database270.

Rating an Electronic Document

Each document record 290 stored in the score database 270 relates to arating a specific member has assigned to an electronic document whetherthe electronic document is a web site, web page or other item ofelectronic content on one of the content servers 265. As a user browseselectronic documents, the user can rate and comment on any of theelectronic documents they care to.

FIG. 5 illustrates a flowchart of a method 500 of a user rating anelectronic document the member has reviewed. In one aspect, method 500is implemented by a plug-in to a browser application on a dataprocessing system 101 being operated by the member. In another aspect,the method may be integrated into the browser application itself. Themethod 500 begins with a member logging into the central server 260 atstep 510. Typically, this involves the user entering a unique usernameand password into their data processing system 101 so that the dataprocessing system 101 can transmit this information to the centralserver 260. The user typically has previously registered with thecentral server 260 and a member record 280 has been createdcorresponding to the to user making the user a member of the member set.The username and password that is transmitted to the central server 260is used to access the member record 280 in the score database 270 thatcorresponds to the user. This allows the central server 260 to determinewhich member from the set of members is accessing the central server260.

The user can then browse electronic documents on various content servers265 at step 520. The electronic documents browsed by the user arecontained on one of the content servers 265. The user can either findthe electronic document he or she is browsing by either accessing thecontent server 265 directly (such as by using a URL or a hyperlink toaccess the content on the content server 265) or by using a searchengine, such as the search engine server 250, as is commonly known inthe art, to locate the electronic content.

The user can continue to browse different electronic documents at step520 until he or she discovers an electronic document they would like torate. In one aspect, when a user rates a specific webpage on a site, theentire site is rated and associated with the rating. In another aspect,when the user rates a specific webpage in a site, the rating onlyapplies to the specific page and not the site itself.

The electronic document can be any electronic content that can beuniquely identified by an address (such as a URL address or URIaddress), i.e. a specific webpage, graphic, picture, file, link, etc.can be rated and/or a comment provided for by the user for thatelectronic document alternatively the electronic document can be awebsite that has a number of webpages all contained on a content server265.

The user then rates the electronic document at step 530. FIG. 7illustrates a screen shot on one embodiment of a web browser allowing auser to rate an electronic document that is a webpage being viewed bythe browser by selecting a drop down menu and selecting a rating from 0through 10, with 0 being the lowest and 10 being the highest.

Referring again to FIG. 5, the user can, optionally, also make a commentregarding the page or content at step 540 by submitting a typed comment.

The rating and optionally the comment is then forwarded to the centralserver at step 550. Referring to FIGS. 2 and 4, at the central server260, if a document object 400 does not exist that corresponds to theelectronic document being rated, a new document object 400 is createdand the title and URL address or URI address of the electronic documentbeing rated is placed in the title field 410 and address field 420 ofthe document object 400, respectively. The central server 260 uses thelog in information provided at step 510 to associate the new documentrating object 405 with the member object 300 of the user who rated theelectronic document. Additionally, a graphical representation of theelectronic content can be stored in the graphical representation field415. A document rating object 405 is created and associated with thedocument object 400 and the member object 300 corresponding to the userthat has provided the rating. The rating is then inserted in thedocument rating field 440 of the document rating object 405 and anycomment provided by the user is inserted in the comment field 450 of thedocument rating object 405. Optionally, a time and/or date the ratingwas made can be inserted in the date stamp field 455.

However, if a document object 400 that corresponds to the electronicdocument being rated already exists, then the existing document object400 corresponding to the electronic document being rated is obtained. Adocument rating object 405 is then created and the rating placed in thecontent rating field 440 of the new document rating object 405. Thecentral server 260 uses the log in information, provided at step 510, toassociate the new document rating object 405 with the member object 300of the user or member who rated the electronic document. If the userprovided a comment or review of the electronic document, the comment isinserted in the comment field 450. Typically, any field in the ratingrecord 290 that is empty is ignored when it is evaluated so if nocomment is provided, the comment field 450 is simply left empty.Optionally, a value indicating a time and/or date the rating was madecan be inserted in the date stamp field 455.

Referring again to FIG. 5, the method 500 ends and the user can log offthe central server 260 or alternatively go back to browsing electronicdocuments on the content servers.

In this manner, a user or member can rate a number of differentelectronic documents and a rating record 290 is created or updated bythe central server 260 in the score database 270. Each rating record 290indicates which members rated the electronic document and what thatrating was along with any comment about the electronic the user mighthave made. As more and more members of the member set rate electronicdocuments such as web pages and other electronic content, more and moreratings are associated with a specific electronic document. In thismanner, a large collection of ratings of an electronic document canaccumulated in the score database 270 with a number of different membersof the member set each subjectively rating each electronic document.

FIG. 6 illustrates a flowchart of a method 600 that varies slightly frommethod 500 illustrated in the flowchart in FIG. 5, in that it alsoallows a user to assign a category to the electronic document that he orshe is ranking. In one aspect, if a member ranks an electronic documentthat has not yet been rated by one of the members in the member set,method 600 is used instead of method 500 illustrated in FIG. 5. In thismethod, steps 510, 520 530 and 540 are the same, however, an additionalstep 635 is added and step 655 is substituted in place of the previousstep 550 so that the member rating an electronic document for the firsttime can assign the electronic document a category of subject matterthat it falls within.

Step 635 has the user identify a category that the electronic documentthat is being ranked falls into. For example, the user may be presentedwith a list of categories they may choose from (i.e. sports, funny,games, technology, cooking, etc.) and the user may choose one of theprovided categories that he or she believes the electronic document mayfall into. Alternatively, step 635 may simply allow a user to enter atext string giving the user free reign to enter a category of theirdefinition. Additionally, the user may be presented with a mix of thesetwo options where he or she may be presented with a list of categoriesthey may choose from in addition to allowing them to add anothercategory if none of the presented categories seems sufficient to them.

At step 655 the information including the chosen category is provided tothe central server and the central server 260 adds the information tothe category field 460 of the document object 400 corresponding to theelectronic document being rated and the rating to the document ratingfield 440 of the newly created document rating object 405 along withassociating the newly created document rating object 405 with the memberobject 300 of the member that provided the rating.

Rating Other Members in the Member Set

Each member in the member set will rate electronic documents based ontheir own subjective opinions of the electronic document. Because eachmember rates the electronic documents subjectively, different users willoften give the same electronic document different ratings. In many casesthese ratings may be similar, but often different members may give thesame electronic document very different ratings. As the members of themember set get to know how specific members review and/or rate content,the other members can rate that member. A member can identify anothermember in the members set and provide a rating of this other member. Howa member of the member set is rated by other members will then affecthow their ratings of electronic documents is factored into the overallrating of the electronic documents.

FIG. 8 illustrates a flowchart of method 800 which allows a member torate another member in the member set. Method 800 comprises the stepsof: logging in 810; searching for a member 820; and rating the member830.

The method 800 begins and the user logs into the central server at step810. The log in typically involves the member providing a uniqueusername and password to the central server to allow the central serverto retrieve the member record from the database associated with themember.

At step 820, the member is able to search the list of members to locatea member they are interested in rating. Typically, the member is able toenter the name of the member in a search field and the central server260 will display possible matches of members that the member can thenselect from.

At step 830, the member can select another member in the member set andrate that selected member. Again, this rating can be of a number ofdifferent types, however, in one aspect it is again a rating on a scalefrom 0 to 10 with 0 being the lowest rank and 10 being the highest rank.Referring to FIG. 3, the rating is then stored in a user rating object350 in a user rating field 360. The user rating object 350 defines theassociation between the member providing the rating and the other memberhe or she is rating. This rating is later used to determine the weightin the weight field 330 of the member object 300.

In this manner, a member of the member set can rate a number of othermembers based on their previous comments and ratings of electronicdocuments or by their personal association with other members. Thisrating is stored in a user rating field 360 of a user rating object 350.For example, if a member likes the comments of another member and agreeswith his or her ratings of one or more electronic documents, the usermember may rare this member highly. Alternatively, a member may dislikeor disagree with another member's comments and/or rating of one or moreelectronic documents and therefore give that member a lower rating.

Determining the Weighted Ratings of the Members

In addition to each member being able to rate other members in themember set, each members ratings of electronic documents are weighted bya weight factor based on how the other members in the member set ratedthat member. And not only are other member's subjective opinions of amember's ratings and/or reviews used to provide that member with his orher own weight factor that will affect that member's ratings ofelectronic documents, the weight factors determined for each of themembers can also be used to weight the member's ratings of other membersin the member set. In this manner, a rating of a member by a highlyrated member of the member set will have more effect on that member'sdetermined weight factor than a rating provided by a relatively lowlyrated member of the member set.

Each member in the member set will have a weight factor determined forhim or her that is based on the ratings of that member by other membersin the member set. These ratings by other members will in turn beweighted based on the rating member's own determined weight factor whichis in turn based on the weighted ratings of that member by othermembers. Referring to FIG. 3, the weight field 330 of a member's memberobject 300 contains the weight factor that has been determined for thatmember based on the weighted ratings of other members in the member set.Each rating of an individual member will be a based on the ratings givenby other members to that user (i.e. the ratings contained in the userrating field 360 of the user rating objects 350) weighted by the weightfactor determined for the rating member (i.e. the value contained in theweight field 330 of that rating members member object 300).

FIG. 9 illustrates a flowchart of method 900 of setting the weightfactors of the various members in a member set. The method 900 comprisesthe steps of: selecting a first member object 910; obtaining all of theuser rating objects associated with a selected member object 920;determining the weight factor for a selected member object 930; checkingif any other member objects need to have a weight factor determined 940and if so, selecting another member object 950 and repeating steps 910,920, 930 for the next selected member object; iteratively determiningweight factors for the member objects until a sufficient approximationof the weight factors has been achieved 960; and ending the method 900.

At step 910 a first member object is selected and all of the user ratingobjects associated with the selected member object are obtained at step920. Referring to FIG. 3, each member object 300 is associated with anumber of user rating objects 350; a user rating object 350 for eachtime another member has rated that member. Each user rating object 350will be further associated with a member object 300 corresponding to themember that provided the rating.

Referring again to FIG. 9, at step 930, method 900 determines a weightfactor for the selected member object which will be stored in the weightfield. Referring to FIG. 3, the calculated weight factor is placed inthe weight field 330 of the selected member object 300. This weightfactor will be based on all of the ratings of the user by other membersin the member set (the values contained in the user rating field 360 ofthe user rating objects 350) with each rating in the user rating field360 weighted by the rating member's own determined weight factor (thevalue contained in the weight field 330 of the member object 300corresponding to the member doing the rating).

In one aspect, the weight factor of a user, w_(y), to be stored in theweight field 330 of the selected member object 300, where there are Nnumber of user ratings objects 350 associated with the member object300, is given by:

$w_{y} = \frac{\sum\limits_{x = 1}^{N}{f\left( {w_{x},r_{x}} \right)}}{\sum\limits_{x = 1}^{N}{f\left( w_{x} \right)}}$

where w_(x) is the weight factor determined for a member that has ratedthe member (i.e. the value contained in the weight field 330 of themember object 300 for that member); r_(x) is the rating that the othermember gave to the user (i.e. the value contained in the user ratingfield 360 of the user rating object 350 associated with that member'smember object 300); and function f(w_(x), r_(x)) is the rating given tothe user by a member weighted with the weight factor assigned to thatmember and taking into account the number of members who have rated theuser.

The specific equation used may be varied depending on the goals andvalues that are desired to be optimized by the equation, such as medianrating, whether the number of members who have rated the user should betaken into account, etc. However, in one aspect, using a rating systemof 1 through 10 the equation used to determine the weight factor, w_(y),could simply be the weighted mean as follows:

$w_{y} = \frac{\sum\limits_{x = 1}^{N}{w_{x}*r_{x}}}{\sum\limits_{x = 1}^{N}w_{x}}$

Referring again to FIG. 9, once a weight factor has been calculated forthe selected member object at step 930, the method 900 checks to see ifthere are more member objects to have their weight factors determined atstep 940. If weight factors need to be calculated for any more memberobjects, the next member object is selected at step 950 and steps 910,920, 930 and 940 are repeated until a weight factor has been determinedfor each of the member objects.

At step 960, iteration is used to repeatedly solve for the weightfactors of the members in the member set. The member set forms a complexnetwork of inter-relations. Because weight factors of a member are afunction of both the ratings of other members in the member set and ofthe weight factors of the members providing the ratings, recursiverelationships between the members are formed. When a new weight factoris calculated or recalculated for one of the members, all of the weightfactors of those members that have been rated by that member are alsoaffected. In this manner, once a new weight factor is determined for amember in the member set, the weight factors of members rated by thatmember will be affected and the weight factors of other members willalso have to be updated using that members newly determined weightfactor. These members recalculated weight factors will also affect theweight factors of any other members these members have rated making itdesirable to update these other member's weight factors.

The steps of the method 900 are iteratively repeated until a sufficientlevel of convergence is reached. What is considered a sufficient levelof convergence will depend upon how precise the final calculated weightfactors of the members is desired.

The first time the method 900 is run by the central server 260 on themember records 280, a seeding value is used for the member weightfactors. Periodically, method 900 is run on the member objects 300 totake into account new ratings of members or changes to the ratings bymembers. Additionally, a member's rating of another member may have afinite period that is used before it is removed to prevent stale ratingsfrom affecting a users rating unfairly. Referring to FIG. 3, the datestamp 370 of a user rating object 350 may be examined periodically toremove the user rating object 350 if a predetermined period of time haspassed since the rating was made.

Determining Scores for Rated Electronic Documents

Once weight factors have been determined for the members using method900 shown in FIG. 9, the ratings the members have assigned to electronicdocuments can be used along with the weight factors of the members whoprovided the ratings to determine a score for the electronic documents.

FIG. 10 illustrates a flowchart of a method 1000 for calculating a scorevalue for electronic documents rated by a member. The method 1000comprises the steps of: selecting a first document object 1010;obtaining all of the associated document rating objects for the selecteddocument object 1020; calculating a score for the document object 1030;checking if more electronic documents need to have scores calculated forthem 1040; selecting the next document object 1050 and repeating steps1020, 1030, 1040 and 1050 until scores for all of the document objectshave been determined.

Referring to FIGS. 4 and 10, at step 1010 a first document object 400representing an electronic document is selected and the document ratingobjects 405 associated with the selected document object 400 areobtained at step 1020.

A score for the document object is then calculated at step 1030. Thescore for each document object 400, which will be contained in the scorefield 470, is a function of the ratings of the electronic documentprovided by the members in the member set (the value in the documentrating field 440 of the document rating object 405) with each ratingweighted by the weight factor determined for the member making therating (the value in the weight field 330 of the member object 300associated with the document rating object 405).

In one aspect, the score, S, for an electronic document, y, is aweighted average of the ratings provided by members and given by theequation:

$S_{y} = \frac{\sum\limits_{x = 1}^{N}{w_{x}*r_{x}}}{\sum\limits_{x = 1}^{N}w_{x}}$

where r_(x) is the rating that the member gave to the page (i.e. thevalue contained in the document rating field 440 of the document ratingobject 405) and w_(x) is the weight factor determined for the memberthat has rated the page (i.e. the value contained in the weight field330 of the member object 300 for that member).

Once a score has been calculated for a selected document object 400 atstep 1030, method 1000 checks to determine if more electronic documentsneed to have a score determined for them at step 1040 and if there are,another document object 400 is selected at step 1050 and steps 1020,1030 and 1040 are repeated for the next document object 400.

Once all the document objects 400 are updated with scores, method 1000ends. In this manner, scores are calculated for electronic documentsthat have been rated by members in the member set.

Generally, method 1000 is repeated periodically in order to keep thescores for the electronic documents, that have been rated by members inthe member set, relatively up to date. Scores for electronic documentschange as members provide new ratings of the electronic document.Additionally, as members rate other members the weight factors of themembers who have rated the electronic documents change, causing thescores for the electronic documents these members have rated to change.Additionally, the ratings provided by members may be removed after aperiod of time to prevent stale ratings from affect the score determinedfor an electronic document. All of these factors will cause a score tochange and periodic updating is needed to keep the scores relatively upto date.

Displaying Rated Electronic Documents

The present method of rating an electronic document, such as a webpageor other piece of electronic content, can be used to enhance the displayof electronic documents. The electronic document can be displayed withthe score that was determined for it as describe herein, so that a userviewing the electronic document can also view the score determined forthe electronic document. The electronic document that is displayed doesnot necessarily have to be the complete document, but can simply be ameta-document describing the actual document, such as a search resultsreturned from a search engine and displayed in a list of search results,or with other meta-documents describing a number of different webpageswith a graphical representation of the webpage showing along side thepage score and any comments made regarding the electronic document bymembers of the member set.

Searching by Category

The present method of rating an electronic document, such as a webpageor other piece of electronic content, can be used be used to rank acollection of electronic documents. A collection of stored electronicdocuments can be ranked using the score determined as described hereinand the electronic documents displayed in an order based on the theirdetermined scores or based on a ranking formula that factors in thescores as part of the ranking formula. Additionally, by having membersin the member set assign a category to an electronic document that theyhave rated, the electronic documents can be sorted by category, with allthe electronic documents assigned to one category grouped together andthen ranked within the category using the score. This allows users tosearch an electronic document collection by selecting a category ofelectronic documents they are interested in and then viewing the rankedelectronic documents.

As a Ranking System or as a Component of a Ranking System in a SearchEngine

The present method of determining a score for rating an electronicdocument, such as a webpage or other piece of content, based on weightedmember ratings from members in a member set can be used in conjunctionwith a search engine's search index. The rating method can be used torank electronic documents located in the search index or as a factor ina larger ranking algorithm used by a search engine to rank the resultsof a search of its index or database. The rating method is incorporatedinto a web search engine to rank located electronic documents takinginto account the ratings. The search engine will locate electronicdocuments that match a search criteria and generate a list of electronicdocuments. This list can then be sorted by the search engine using thescore calculated for the electronic documents using the rating methoddisclosed above with the high ranking electronic documents listed firstand lower ranking electronic documents listed later. The rankings can beaccomplished using solely the score determined using the rating methodsdisclosed or with the score as one of a number of factors as part of alarger ranking algorithm.

To Reorder a List of Search Results

The present method of assigning a score to a document can also be usedto reorder a list of search results. A search conducted by a third partysearch engine or other search service typically returns a list of searchresults that have been ranked by the search engine or other searchservice itself. By using the score determined as described herein theselists of search results can be reordered by re-ranking search resultsthat have been rated by members in the member set.

The foregoing is considered as illustrative only of the principles ofthe invention. Further, since numerous changes and modifications willreadily occur to those skilled in the art, it is not desired to limitthe invention to the exact construction and operation shown anddescribed, and accordingly, all such suitable changes or modificationsin structure or operation which may be resorted to are intended to fallwithin the scope of the claimed invention.

1. A method for assigning a score to an electronic document, the methodcomprising: assigning a weight factor to each member of a plurality ofmembers; receiving a rating value of an electronic document from each ofat least two of the members of the plurality of members; and determininga score for the electronic document, the score based on each ratingvalue assigned to the electronic document by a member with each ratingvalue being weighted by the weight factor assigned to the member whoprovided the rating.
 2. The method of claim 1 wherein the weight factorassigned to a member is based on user ratings assigned to the member byother members in the member set.
 3. The method of claim 2 wherein eachuser rating used to determine the weight factor for a member is weightedby the weight factor determined for the member assigning the userrating.
 4. The method of claim 3 wherein the weight factors for themembers are determined by iteration before the score is determined. 5.The method of claim 4 wherein the electronic document is on theInternet.
 6. The method of claim 5 wherein the electronic document is awebpage.
 7. A computer readable memory having recorded thereonstatements and instructions for execution by a data processing system tocarry out the method of assigning a score to an electronic document, themethod comprising: assigning a weight factor to each member of aplurality of members; receiving a rating value of an electronic documentfrom each of at least two of the members of the plurality of members;and determining a score for the electronic document, the score based oneach rating value assigned to the electronic document by a member witheach rating value being weighted by the weight factor assigned to themember who provided the rating.
 8. A computer readable memory of claim 7wherein the weight factor assigned to a member is based on user ratingsassigned to the member by other members in the member set.
 9. A computerreadable memory of claim 8 wherein each user rating used to determinethe weight factor for a member is weighted by the weight factordetermined for the member assigning the user rating.
 10. A computerreadable memory of claim 9 wherein the weight factors for the membersare determined by iteration before the score is determined.
 11. A dataprocessing system for assigning a score to an electronic document, thedata processing system comprising: at least one processor; a memoryoperatively coupled to the at least one processor; and a program modulestored in the memory and operative for providing instructions to the atleast one processor, the at least one processor responsive to theinstructions of the program module, the program module operative for:assigning a weight factor to each member of a plurality of members;receiving a rating value of an electronic document from each of at leasttwo of the members of the plurality of members; and determining a scorefor the electronic document, the score based on each rating valueassigned to the electronic document by a member with each rating valuebeing weighted by the weight factor assigned to the member who providedthe rating.
 12. The data processing system of claim 11 wherein theweight factor assigned to a member is based on user ratings assigned tothe member by other members in the member set.
 13. The method of claim12 wherein each user rating used to determine the weight factor for amember is weighted by the weight factor determined for the memberassigning the user rating.
 14. The method of claim 13 wherein the weightfactors for the members are determined by iteration before the score isdetermined.
 15. The method of claim 14 wherein the electronic documentis on the Internet.
 16. A memory for storing data for access by at leastone application program being executed on a data processing system,comprising: a data structure stored in said memory, said data structureincluding information resident in a database used by said at least oneapplication program and comprising: a document record associated with anelectronic document and having a score value; and at least two memberrecords each member record representing one of a plurality of members,each member record having a weight factor and associated with thedocument record by a rating value assigned to the electronic document bythe member represented by the member record, wherein the score value ofthe document record is based on each rating value assigned to theelectronic document associated with the document record and with eachrating value weighted by the weight factor of the member recordassociated with the rating value.
 17. The memory of claim 16 wherein theelectronic document is located on the internet.
 18. The memory of claim17 wherein the document record is associated with the electronicdocument by a URL address, identifying the location of the electronicdocument on the internet.
 19. The memory of claim 18 wherein theelectronic document is a webpage.
 20. The memory of claim 16 wherein theweight factor of each of the at least two member records is based on auser rating assigned to the member represented by the member record byat least two other members.
 21. The memory of claim 20 wherein each userrating is assigned to the member represented by the member record isweighted by a weight factor of a member record represented by the memberproviding the user rating.
 22. The memory of claim 21 wherein the weightfactors of the member records are determined using iteration before thescore value is determined.